In this study, we conduct an empirical analysis of the performance of five popular data mining methods--neural networks, logistic regression, linear discriminant analysis, decision trees, and nearest neighbor--on two binary classification problems from the credit evaluation domain. Whereas most studies comparing data mining methods have employed accuracy as a performance measure, we argue that, for problems such as credit evaluation, the focus should be on minimizing misclassification cost. We first generate receiver operating characteristic (ROC) curves for the classifiers and use the area under the curve (AUC) measure to compare aggregate performance of the five methods over the spectrum of decision thresholds. Next, using the ROC results, we propose a method for tuning the classifiers by identifying optimal decision thresholds. We compare the methods based on expected costs across a range of cost-probability ratios. In addition to expected cost and AUC, we evaluate the models on the basis of their generalizability to unseen data, their scalability to other problems in the domain, and their robustness against changes in class distributions. We found that the performance of logistic regression and neural network models was superior under most conditions. In contrast, decision tree and nearest neighbor models yielded higher costs, and were much less generalizable and robust than the other models. An important finding of this research is that the models can be effectively tuned post hoc to make them cost sensitive, even though they were built without incorporating misclassification costs.
This paper presents an integrated and comprehensive framework for decision support. A model integrating case-based reasoning with constraint posting and multicriteria decision making is proposed for providing effective and efficient assistance in solving routine design problems. The model is developed based on an analysis of the knowledge acquired from experts in engineering design, and is subsequently operationalized as a computer-based design assistant called IDEA. IDEA employs constraint posting to initially bound the design space and to maintain consistency of the design solutions. Case-based reasoning allows IDEA to generate new designs by retrieving, adapting, and composing from similar cases in memory. Finally, IDEA optimizes multiple objectives to identify a set of pareto-optimal designs. By organizing computer memory as a collection of cases and case snippets, and by adapting and synthesizing those cases and snippets-using techniques similar to those employed by design experts-IDEA provides valuable design assistance. In addition to providing a framework for decision support, the research makes specific contributions to case-based design. It shows how case snippets can be retrieved, adapted, and synthesized to generate multiple design solutions, whose consistency is enforced through a dynamic constraint management mechanism. The concepts and techniques developed for performing dynamic adaptation (adaptation during composition from case snippets) and for maintaining an evolving solution space (a solution space that shrinks and expands over time) contribute to the state-of-the-art in case-based design.
Requirements modeling constitutes one of the most important phases of the systems development life cycle. Despite the proliferation of methodologies and models for requirements analysis, empirical work examining their relative efficacy is limited. This paper presents an empirical examination of object-oriented and process-oriented methodologies as applied to object-oriented and process-oriented tasks. The conceptual basis of the research model is derived from the theory of cognitive fit, which posits that superior problem-solving performance will result when the problem-solving task and the problem-solving tool emphasize the same type of information. Two groups of subjects participated in an experiment that required them to construct solutions to two requirements-modeling tasks, one process-oriented and the other object-oriented. One group employed the object-oriented tool while the other used the process-oriented tool. As predicted by the theory of cognitive fit, superior performance was observed when the process-oriented tool was applied to the process-oriented task. For the object-oriented task, however, the performance effects of cognitive fit require further investigation since there was no difference in subject performance across the two tools.